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■ABSTRACT 

" / Normal curve equivalent achievement gains estimates 

■were compared with RMC Title I evaluation Models Al and El. The 
comparison focused upon the amount of lias introduced by Model Al 
when its underlying assumptions were violated. The ircdel assumes^ 
•first, that the local school populaticr is accurately represented by 
the national norm group; and secondly, that the percentile standing 
o^^ treatment group on the pretest remained unchanged on the 
posttest in the abscence of treatment effect. Data were from a Model 
B1 evaluation — kindergarten children were pre- and pcsttested with 
.the Aural Comprehension subtest of the Stanford Early School 
Achievement Test, while the first and second grade children were pre- 
.,and posttest ed with the Woodcock Reading Mastery Test. Comparison of 
the three model B1 gain estimates (unadjusted, covariance adjusted, 
and principal -axis adjusted), illustrated the wide variation between 
estimates. For kindergarten groups, the adjustments were relatively 
modest, but for grades one and two, the adjustments caused a dramatic 
shift from negative to positive estiw ates. Mcdel Bl (adjusted) 
provided much lower estimates for kindergarten and much higher 
estimates for grades one and two than did Model Al. Questions were 
raised concerning representativeness of the norm grcup and validity 
of the equipercentile assumption. (Author/CP) 
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Introduction 

Several local educate- I wicies iiave now ImplenenEed one or mass, oi 
the RMC models for Ticle I evaluBtion. The three mdels are referred, to 
as: **llodel A, the Ncrw-Pe£exeaccd Model; Model B, the Conrrol Grout) 
Model; and Model C, the &«cci^ Stegressim Model" dallawdge and W od, 
1978, p« 30). The oirdcr raf: pz^Ssrence (±n terms ox scl «rtrff lc rigcir) is 
given as Model B, Model C. aaidiiloodel A rTallmadge and Waod^ -pp. Z2-^2ir)^ 
This paper reports the res-: Its Tsf a stmsr coffiparinp- HCE ^aln esriit :adi 
using Model A procedures wr=?r ^^gCE gain sstimated m-±B^ Model 3 pre -i^iteres. 
(Technically, the compa=±»u becwees .^o6sl Al ail3 ^fimae^ Bl tariiSh 
reqn±res norm-ref erencee^ Xesos^^as oppswd to Moi^-^^ A2 and B2.^iy^ii 
allow f^fc: criterion-rei ±arfc t^Kr±n?). 

^^If. pasic different tbti-y«r mDdels ^ "byv Om.sxpec&ed xxMm^mnt 
m^^jvt^ 3 derived, im^^ a ^^mts^-sffffg tCE aonD^^sanp frcf^a nacftDMiUy 
^'"':m^^^^^s^ nest as a sonH '^ %mo mtd jwwuimes ""^Kt the "ssKmoicla.e 
.atftLn^ ' pSsB treatmen*- p^-^- on a pretest wocV'id,-±n the absence of 
?J*men effect , rema:*a Taffi**tt^^d on a pcattesr fgsllmadge and Wood>. • 
UpdaJU B -ssarrly uses thsL i-aut ^t^est percentile standing (conveirted to NCEs) 
t^'i the ccntrol group for wtit attp«cted no-treatnentt estimate (with 
(3ir vlthout adjustment). 

At this point, it is ^aatti. j apparent, that- tarn: Model Al and Model 
^ CD yield equivalent NCE ada estimates, tarn [■■■■imptioM are critical. 
Ihe first assumption is that tiie local school pcopoftation is accurately 
xcp esented by the norm grcmt pcopulation in tenaa of relevant character- 
ist$*es (e.g. SES, ethnic, uraan rural). The s«csai^ assumption, already 
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stated^ Is that the percentile s^andibos of the tsassssnt grotD on the 
pretest remain undaifsed on the posttstt Im the abbonice of treatment 
effeoL. Clearly, in i field s^tlnf;, tiie £irst a^snoiption viXl be 
vlolfffflRiL ^o some extM^ and tte aecoad a88unapt±on has been qustioned 
under «sr circumstant^^s (Linn, 1978> pp« 8«-12). Thngy given a properly 
isspleaoced control s^oixp .design, sixv^ as Hodel B, imiempirica^ comparison 
of Title I NCE gaiiK Lnwl e d by appiiylng bcoch Mod£&^ Al and Mwrtel B.1 
procedures should psr-^lizie 21 Jsi^ltsiwloe of tSie iammft of bias i^t ^del 
Al procedures can ittcromrgs: i&. Che esC^Jiwte of Xij=^ I NCE galn« 

The key point upon waiidi t^sisB copipara^ve study rasts is "a properly 
Implemented control group^*" Udeal:/^ for Hodel Bl, thdia requiraas rmidam 
assignment to treatment 2sed entail groupis* S±2ce random assj^me.nr: Is 
frequently not ^esslble^ '"ratobom ±tx effect*^ has beesi fnirmed accassahLe 
(Tallmadge and Wood, p. 6). Ixi sltuatioas iTfaere pnt&ft^ treal iiiwi 11 111 if rol 
group differences exist, tm acjustmmr procedures (analysis of uuv^ rla nce 
and principal-axis:) are psxrLfed C^allnsidge and Homt, 1976) • Tlie 
selection of adjustment procecair« depeods upcn wfaes^Ksr- or not it can h& 
assumed the two groups are saaDPles from clsa same pBomlatlon* If the tvc 
groups are from die same popiul%t;ton, covar^^ce aid^Mtscment ±s appropriate; 
while i£ the two groups are f=3« < if fgremt jopulatiilM:, prtndpal-axls ar 
standardized change score adjnstaK^ is at^fpcoprlaite (Tallmadge and Horst, 
pp. 4-11; Kenney, 1975, pp. 345-3.61; llaa %ad Werts, 1977, pp. 229-234-) 
In any case, Tallmadge and Horsr caudoa ticat tne adjustmmts are not 
appropriate if the treatment group aat mntr ol group regression lines 



v(or principal-axes) are not parallel such as might occur If the two 
flnmnn exhibited different growth rates* Keeping these cautions in mind» 
the following sections: describe the data source and selection of 
contrw»r groups ; describe the methods for analysis; present the results 
ami tfLscussion thereof; and present the summary and conclusions of the 
stsdy* 
Data Source 

The Schaumburg (Illinois) School District 54 implemented a Model Bl 
evaOLuatlon plan to evaluate Title I programs in kindergarten, first, and 
second grades* The comtrol groups consisted of children from "comparable" 
but nonparticlpating schools within the district* Identical procedures 
were followed in selecting Treatment and Control group children and all 
children * irere selected by the Director of Research through review o£ 
inrevlous testing and/or recommendation of Child Study Teams* From the 
**pool" of pupils eligible for participation. Title I participants were 
selected solely on attendance at a school having Title I programs* 
IZontrol group children were pre- and posttested by substitute teachers 
trained for the purpose, while treatment group children were tested by 
classroom teachers* 

Kindergarten children vera pre- and posttested with the Aural Comprehension 
subtest of the Stanford Early School Achievement Test (SESAT) while first 
and second grade children were pre*- and posttested with the individually 
administered Woodcock Reading Mastery Test (WRM)* Raw score (RS) data were 
converted to NCE data through percentile norm tables for the SESAT and 
first grade WRM tests* For the second grade WRM test, raw scores were 
converted directly to NCEs using the newly developed 1978 norms* 



Because both treatment group and control group children also received 
assistance from a learning disabilities specialist (ID) and/or a speech 
therapist (ST) and there was some concern that the children in the control, 
group might have received substantially more assistance than treatment 
group children, Individual pnrrf.l data in terms cnf total minutes spent wltfa. 
each therapist were also collected* 
Method for Analysis 

To coH^are Models Al and Bl, NCE gains were estimated acconSlng to 
the procedures recommended for each model. Specifically, for Model Al, 
gains were estimated according to the formula: 

NCE^ » NCE2 - NCE J (1). 

For Model Bl, both unadjusted and adjusted NCE gains were estimated 
according to the formulas: 

NCEg « NCE^ - NCE^ no adjustment (2) ; 

NCEg^ - [NCE2 - b(NCE^^j - NCE^J"^-^^) ] - [nCE^ - b(NCEj - NCE^^^^)] 

covariance adjustment (3); 

SD 

NCE„ = NCeJ - [nCeJ - (NCeJ^ - NCE^ 1 • principal- axis adjustment (4 

Bp 2 A SD. ± . J. 



The notatxdmz enm^srjmd In riiwiiiiiii -1-4 Is given i^ov: 

NCE ^rtfth an A^nr B^ct'^^^. Bp) subscript re£^ to aeaxi gs±n 
est±MBed.:2ca9c±Lng^ ^odel Al or Model BI parocednses 
resjMKtrtvEEiy • 

T or X sBUfsrsciimft rftf^^^t to Treatment Group :3r Control 
J&roupr^aMxxs tam^cOBaBtL 7 ^ C ^ref ers tn tbm overall 
wan) 

1 or 2 «i£scrttpt9 r&ier :±t IIEIE pretest (1) or^NCE posttest 
(2) mesB* 

b is ±9«s pool^i^ wirfcts groups regression coefficient of T^CEj ^ 
NCEj- 

SD2 ^sad SD)- th^T iDoled within group post- 

and ^eteat srsmdsr^ deviations respectively. 

Results 

NCE moB: anc standards deviations are reported in Table 1. Significant 
dlf ferensef < betweeai groups as determined by exact (s: approz±mate 

when requtrawc ) t-tidsts are also noted in Table !• There were Tiis> significant 
differencfie^ ^^weesc the kindergarten (Kg) groups on any of the vasures. 
First grad^ gxoups differed significantly on both pretest (RS and 

NCE) meaaujaiii^ Sascond grade (F2) groups differed significantly on pre- 
and postte0C;^F^aTid NCE measures. Within grade level, none of th* treatment * 
control group ::^0mparisons on time spent with speech therapist or learning 
disabilitiefli tasKfaeer were significant (however, the variances wene quite 
disparate ix:^ some cases) • 



Tables 2A» 2B, and 2C report The vlthln group correlations for each 
of the three grades (Sg^ Pl» and TZJ:* Also reported ars the groiq)-var±at» 
correlations # Trestmnt Group cortElLsrlons are glvssi below the diagonal 
and Control Group cot£te±a±lons ace: g^ron above the diagonal. These 
results are Ineiiided *'t>. assist In fiTsrosslon of the liCE gain estimates 
derived from ttMftiel All aod Model Bl ^octseiures • 

NCE gain oKtiaaBts are given ^sxr <«sh of the models In Table 3* 
Within group sui^p^ s fzrr the two aijast^MOt procedures suggested for Model 
Bl are also par ic^ 
Discussion ogSsBu: s 

The disc ssligg is focused on the Jodel Al and Model Bl NCE gain 
estimates gl%BC In. arable 3. However references will be made to the 
other tables tc as&-4at In Interpretl'-::^ the differences between NCE gain 
estimates. 

ComparlaBLi of the three Model £1 gain estimates given In Table 
3 lllustrates^ the wide variation becween estimates depending on whether 
or not adjuscfid means were used In^computlng the NCE gain. For Kg 
groups 9 the adjustments were relatively modest but for PI and P2 groups 
the adjustments provided a dramatic shift from negative to positive 
estimates of project Impact. The Kg results point to the problems of 
assessing project Impact at that age with these models; not surprisingly 
the pre-posttest correlations » although significant » are very low^ on the 
order of .3 (see Table 2A). The PI pre- posttest correlations^ are 
somewhat better on the order of .6 (see Table 2B) while the P2 pre-post- 
test correlations are quite respectable^ on the order of .8 (see Table 2C). 



Examination of the within group regression slopes indicated Tia-atmesit and 
Control groups had similar slopes. F-tests for heterogeneous i^opes 
(Winer, 1971, p. 773) xe^aled no significant differences (Kg^_F]_^i48 < 

1; ^1,148 - 1-5; n,107 < !)• 

Nov changing focas, examine Model Al NCE gain estimates f'ved 
for both Treatment and Control groups. Under the Model AL equenyercen- 
tile assumption, Contxal Group NCE gains should be close to zera vhich 
clearly is not the case* Since the differences between Trea^eat and 
Control groups on time spent with either speech therapist or lisKrxxing 
disabilities teachers were not significant ( a> .05) it is untllfteiy that' 
these additional services account for the Control group NCE igaSaa estimates 
derived under Model Al, presumably, the additional services :ran be 
considered Part of the regular school program. 

Since Model Al is highly dependent on appropriate te8t±ng (e.g. 
test and level selection, empirical norms, or testing within the empirical 
norming period) some of the Control Group nonzero Model Al gains, may in 
fact have been due to testing problems. Both Kg and PI tests insaolved i^t \.T3o- 
lated norms, spring for the SESAT (Kg) and fall for the WRM (old norms, 
PI). Where fall norms are interpolated, a positive bias is likely to be 
introduced in the NCE gain estimate (Horst, 1978, p. 168); presumably use 
of spring interpolated norms would result in the reverse, a negative bias. 
However, the positive Kg Control Group gains (spring interpotated normB) 
and the negative PI Control Group gains (fall interpolated norms) would 
probably not account for the large nonzero ;^alns demonstrated by the Kg and PI 
Control Group Model Al gains. P2 Control Group Model Al eains were also Islv^b 
and negative (fall and spring norms were empirical). Another possibility at 
least for P2 (and probably for PI) is that the norms 



Introdtice considerable <ii2Es. Strand, Anderson and Sauer (1979, p 15) 
found that the pre* andi ^smsBEZest empirical norm group raw score means 
translated to NCEs of 533 ^nuL 46.9 respectively resulting In an NCE gain 
of -6,1 for the P2 norm groaip. Assuming that the median pre- and posttest 
raw scores are close to t±e mean raw scores (or that the distributions are 
normal) , then the P2 ^RBH norms appeal to have a negative bias on the order 
of -6.1; which when subtracted from the Model Al Control Group NCE gain 
results In an NCE galm of -2.6, much more In keeping with an expected zero 
gain for the Control irtxup. Finally, Linn cites a study by Kaskowltz and 
Norwood which suggested that where pretest scores are not particularly low, 
posttest expected scores will be Inflated, (Linn, 1978, p. 11). This also 
might have Influenced the Control Groiip Model Al gain estinates since the 
pretest scores were not particularly low. 

Comparing Treatment group NCE gains estimated under Model Al with 
those estimated under Model Bl (adjusted) indicates Model Bl provided a 
substantially lower NCE gain estimate for Kg and substantially higher NCE 
gain estimates for PI and P2 than the corresponding Model Al estimates. An 
oddity apparent for the three grades is that the difference between Treatment 
and Control Group's Model Al estimates is very close to the Model Bl 
principal-axis or covariance adjusted estimate of Treatment Group gains. 
This result needs further investigation since it may provide a basis for 
simplifying Model Bl procedures. 
Summary and Conclusions 

This study compared Model Al NCE gain estimates with Model Bl NCE 
gain estimates. The comparison was intended to indicate the amount 
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of bias Introduced by Model Al procedures when Its underlying assumptions 
were violated. The data were from a Model Bl Title I evaluation conducted 
In the Schauiid>urg (Illinois) District 54 school system for grades Kg, Pi 
and P2. 

Results were equivocal with Kg Model Al NCE gains considerably 
larger than Model Bl NCE gains with the reverse ocurring for 

PI and P2 NCE gain estimates (adjusted Model Bl NCE gains were greater than Model 

Al NCE gains). Some questions were raised concerning representativeness 

of the norm group and validity of the equlpercentlle assumption, though 

not in the usual direction with respect to the equlpercentlle assumption. 

The NCE pretest means were somewhat high, especially for the Control 

Group (PI and P2 grades) which may have reverse impllcatons with respect 

to the equlpercentlle assumption. 

Test bias was mentioned as a probable problem area. Comparisons of 
Treatment and Control Group means on Model Al gain estimates can provide some 
insight into the size and direction of the bias. Test specific bias can 
be partially examined through comparison of Norm Group raw score means 
converted to NCEs. The latter type of bias may "cancel" out in large 
aggregates across tests, but for within dl^^trlct use of data this becomes 
an important consideration, especially if the bias is different across 
levels of a test. 

The study is limited in that comparisons for only grades Kg, PI and 

P2 were possible. Since the KMC models were Intended only for grades 

P2 and above, this is a rather severe restriction. 
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A further limitation Is that data from only one district were 
examined. It Is quite apparent that results from several studies 
across grades and tests should be examined before any pattern would 
be dlscemable. 

Since any Model Bl evaluation data can also be applied to ^del 
Al procedures If a few precautions are taken (e.g. testing during 
appropriate times, with respect to normlng dates), additional empiri- 
cal studies could be conducted with relative ease. 

Possibly, some simulation studies would be Informative by allow- 
ing for planned variation In pretest means and ranges of differences 
between Treatment and Control groups. Certainly parameters affecting 
bias In Model Al need further refinement, especially as Model Al Is 
likely to be the most frequently implemented Model. 
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TABLE 1 



TREATMENT(T) AND COSTROL (C) GROUP RAW SCORE (RS) AND NCE 
MEANS AND STANDARD DEVIATIONS (SD) FOR KINDERGARTEN (Kg) 
FIRST (PI) AND SECOND (P2) GRADE PUPUILS 



Time (Mln.) With 

PreRS PostRS Pre NCE Post NCE Speech Learning 
Group N Mean (SD) Mean (SD) Mean (SD) Mean (SD) Thewplst Dlsabllltlea Twcher 
Mean (SD) Mean (SD) 

T 82 13.5 (3.49) 19.1 (3.37) 39.0 (15.41) 47.0 (14.82) 106.9 (226.16)^^^ 4^^^^^^^^^ 

C 70 14.6 (3.63) 19.6 (4.11) 43.9 (16.28) 50.0 (18.98) 146.6 (294.76) 50.9 (176.15) 



PI T 83 39.2 (5.42)^ 69.2 (14.37) 48.2 (10.20)^ 36.3 (20.78) 116.0 (259.66) 113.5 (287.63) 
C 69 43.3 (7.08) 70.4 (12.51) 55.9 (12.87) 37.8 (18.13) 166.0 (246.12) 139.0 (345.90) 



P2 T 46 77.9 (38.51)^ 149.7 (50.00)^ 34.2 (19.58)^ 35.8 (19.76)* 42.6 (143.46) 177.4 (648.93) 
C 65 127.3 (46.01) 175.1 (39.79) 54.1 (15.85) 45.4 (16.13) 71.6 (171.45) 101.1 (264.61) 



%<.01 
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TABLE 2A 

KINDERGARTEN PRE-POSTTEST CORRELATIONS (RAW SCORE AND 
NCE) AND CORREUTIONS FOR TIME SPENT WITH A SPEECH 
THERAPIST (ST) OR LEARNING DISABILITIES TEACHER 
(LDf 



t ■ 
J . ■ 

■ 

'y. ' 


Grouo^ 


PreRS 


PostRS 


PreNCE 


PostNCE 


Time Time 
w/ w/ 
Speech Learning 
Therapist Disabilities Teacher 


% , ■■ 

PreRS 


.150 




.373^ 


.992^ 


.345 


.017 


.040 


BostRS 

r.' ■. ■ 


.068 


f% Old 

.322 




,355 


• 992 


.162 


,109 


?reNCE 


.152 


.996^ 


.331^ 




.327^ 


.041 


.043 


fostNGE 


.090 


.323*^ 


.997^ 


c 

.332 




.154 


.087 


ST 


,076 


-.060 


-.183*^ 


-.052 


-.176 




-.016 




.035 


.049 


-.004 


.076 


.001 


.225^ 





^The correlations within the square section as marked 
are within group correlations; Treatment Group 
correlations are given in the lower half of the , 
matrix while Control Group correlations are given 
in the upper half of the matrix. 

^The correlations under the column heading "Group" 
are correlations between group membership (treatment 
or control) and the variables given as row headings. 

^a<.05 
^o<.01 
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TABLE 2B 










^v' ■ 


FIRST GRADE PRE-POSTTEST COKREUTIOHS (RAW SCORE AND 
NCE) AMD CORRELATIONS FOR TIME SFESir VITH A SPEECH 
THERAPIST (ST) OR LEARNING DISABILITIES TEACHER 
(LD) 

« 




Groups 


PreRS 


PostRS 


PreNCE 


PostNCE 


Time 

w/ 
Speech 
Therapist 


If! J _ 

Time 
w/ 

Learning 

Disabilities Teacher 


SreRS 


.313^ 




.626^ 


.998^ 


.627^ 


-.060 


-.041 




.045 


.575*^ 




, d 
.631 


1.000° 


-.182 . 


-.207'^ 


iPreNGE 


.319^ 


.998*^ 


.578^ 




.631^^ 


-.070 


-.040 


I-- 

iostNCE 


.040 


.572^^ 


.999^ 


.575^ 




-.185 


-.207^ 


St 


.098 


.095 


.079 


.091 


.078 




.066 




.040 


-.176 


-.223'^ 


-.180 




,268*^ 









^he correlations within the square section as marked 
are within group correlations; Treatment Group 
correlations are given in the lower half of the 
matrix while Control Group correlations are given 
in the upper half of the matrix. 



, The correlations under the column heading "Group" 

are correlations between group membership (treatment 
or control) and the variables given as row headings, 

f: ^a<.05 

I'/ , 

t. ^o<.01 

P'[ 

V 
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TABLE 2C 

SECOND FRE-FOSTTEST CORfi&ATIONS (RAW SCORE AND 
to) -AID CORRELATIONS FOR TDK SFENT WITH A SFEECH 
IBERAEEST (ST) OR LEARNING DISHILITIES TEACHER 
(LD)^ 



Group 

,495^ 
.254^ 
.492^ 
,260** 
.089 
-.085 



PreRS PostRS PreNCE 



Time Tine 

w/ w/ 

PostNCE Speech Learning 

Therapist Disabilities Teacher 



.886^ 



.825 
.950** 
.832** 
.336*^ 
-.076 



.982' 



.883' 



.824' 
.992' 
.123 
-.272' 



.883 



.997 



.885 



.826' 
.208 



..223'' 
-.150 
-.273'^ 
-.149 



.155 



-.373 



-.363' 



-.404'- 



-.386' 



.057 



-.136 -.193 



.025 



a 



The correlations within the square section as marked 
are within group correlations; Treatment Group 
correlations are given in the lower half of the 
matrix while Control Group correlations are given 
in the upper half of the matrix. 

The correlations under the column heading "Group" 
are correiatious between group membership (treatment 
or control) and the variables given as row headings. 



'a<.05 



a<.01 
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TABLE 3 



MODEL Al m MODEL Bl NCE CUN ESTIMATES 

FOR HNDERGARTEN (Kg), FIKBI (PI) AND SECOND (P2) 

GRADES 



Grade 


KCE Gain 
Model Al^ 
(Formula 1) 


Unadjusted^ 
(Formula 2) 


NCE Gain Model Bl 

Covarlance^ 
Adjusted 
(Formula 3) 


Principal-axis^ 
Adjusted 
(Formula 4) 


^ 


8.0 (6.1) 


-3.0 (4.9) 


-1.3 (.32, 


.38) 


2.2 (.96, 1.17) 


Pi 


-11.9 (-18.1) 


-1.5 (7.7) 


6.2 (1.17, 


.89). 


12.0 (2.04, 1.41) 


P2 


1.6 (-8.7) 


-9.6 (19.9) 


7.6 (.82, 


.90), 


10.8 (1*01, 1.02) 



Given in parentheses are the Control Group gains determined 
according to Model Al procedures. 



Given in parentheses are the pretest NCE differences between 
the Control and Treatment groups (NCeJ - NCE J ), 

'Given in parentheses are the within group regression slopes 
for Treatment and Control Groups respectively. 

Given in parentheses are the within group principal -axis slopes 
for the Treatment and Control Groups respectively. 
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